Same goes for Metal on MacOS. At the moment metal is usable on some pre Apple Silicon Macs (amazingly actually) and on some it just crashes or hangs allocating kernels. Maybe not worth an effort though

Tks @slaren
Please reply to these questions before I retouch this PR and/or prepare the PR in llama.cpp.

I am not convinced about the approach in the CUDA backend. It will require a lot of changes to change every single CUDA_CHECK. I would consider changing CUDA_CHECK to throw an exception instead, and catching them in the ggml-backend functions. The ggml-backend functions must never leak exceptions, so consider adding a noexcept to all the ggml-backend interface functions when building from C++. This will also require ensuring that every resource is allocated via RAII in an exception-safe manner.

I m ok to retouch the way you prefer it but please be precise in order to save time for everybody.
Do you propose something like:

void ggml_cuda_error(...) {
  ...
  if (THROW_ON_CUDA_ERROR)
    throw std::runtime_error(...);
  else
    GGML_ABORT(...);
}

?
Tks

Since a backend cannot allow exceptions to escape (to allow dynamic linking at the backend API boundary), it seems we only have the question of which backend functions need modification to allow non-abort handling of allocation failures. Is it really only ggml_tallocr_alloc and init_tensor, @WilliamTambellini?
I suggest documenting in the new API that the backend is still usable after such a return (i.e. immediately retrying with smaller input is permissible).
Clearly there are many aborts in ggml generally (and the cuda backend arguably) that should stay as aborts in this PR.
@slaren is correct that provisions to clean up on OOM return (formerly abort) would need to be verified and/or created before the cuda backend can correctly return an OOM error instead of aborting.
slaren's idea that exceptions could be used internal to the cuda backend makes sense (and that this implies that cuda backend internal cleanup provisions are RAII+exception safe).
I'm not sure about slaren's suggestion of making every CUDA_CHECK throw an exception, as this increases the number of code paths that need to be made exception safe, considering the overall purpose of this PR is just to support orderly error return for OOM conditions. But if ggml veterans think that there will be additional recoverable-error conditions then this more general approach could be a decent investment.

Yes, that's what I am proposing. Throw an exception in the CHECK macros in case of failure, and catch them in the ggml-backend functions that can fail to return an error to the caller.

I'm not sure about slaren's suggestion of making every CUDA_CHECK throw an exception, as this increases the number of code paths that need to be made exception safe, considering the overall purpose of this PR is just to support orderly error return for OOM conditions. But if ggml veterans think that there will be additional recoverable-error conditions then this more general approach could be a decent investment.

My opinion is that we should only abort when some pre-condition that is expected to be met by the caller is not. These are programming errors that indicate that ggml is not being used correctly, and usually can be fixed easily. However we should never crash the application just because a CUDA function returns an error - we must always provide applications some way to recover from this, or at least give it a chance to shut down cleanly.

Yes, that's what I am proposing. Throw an exception in the CHECK macros in case of failure, and catch them in the ggml-backend functions that can fail to return an error to the caller.

hum, your reply is confusing: my question was about retouching the ggml_cuda_error() fn but you are speaking about the CHECK macro.
Again, please lets be precise: are you speaking about:
CUDA_CHECK_GEN(...) ?
CUDA_CHECK(...) ?
Tks

I mentioned the CHECK macros because that's what the code uses to check CUDA calls, ggml_cuda_error is just an implementation detail of the CHECK macros. All the CHECK macros call ggml_cuda_error, it's ok to throw the exception from it.

Ok, the scope of "all CUDA errors" makes sense, and @slaren is of course correct that inserting a throw in the CUDA_CHECK macros to be caught at or before the backend API level (along with making all users of the macro and their callers exception-safe) would achieve this.

@slaren we are perhaps moving forward although I would prefer not to extend the scope of that PR to all cuda errors. Now, if you tell me precisely what you prefer, perhaps I could do it.
Do you propose something like:

void ggml_cuda_error(const char * stmt, const char * func, const char * file, int line, const char * msg) {
    int id = -1; // in case cudaGetDevice fails
    (void)cudaGetDevice(&id);

    GGML_LOG_ERROR(GGML_CUDA_NAME " error: %s\n", msg);
    GGML_LOG_ERROR("  current device: %d, in function %s at %s:%d\n", id, func, file, line);
    GGML_LOG_ERROR("  %s\n", stmt);
#ifndef __CUDA_ARCH__
    throw std::runtime_error(msg);
#endif
}

?

Yes. To summarize:

• Throw exceptions on CUDA error
• Catch them in all the ggml-backend interface functions that call CUDA functions that may fail
• In cases where returning an error from the function is currently not possible without significant changes to the ggml-backend interface, make it abort anyway after catching the exception